Managing FreeBSD at scale

It is common for sysadmins to jump straight to cloud providers if immediate scale is required. This unnecessarily reduces autonomy and choice, ceding control over many important components to large corporate providers, such as Amazon or Rackspace. While "the cloud" remains an option, sysadmins should strive to maintain full openness on their systems, avoid vendor lock-in, and regain control of infrastructure deployment. This talk presents a "full control" look at managing multiple simultaneous FreeBSD deployments around the globe, independently sourced, yet centrally managed.

Unlike many common deployments, most of our nodes are physical, rather than virtual, and many are on rented machines where we have little control over the selection of hardware and components.

This talk will also cover a number of tools and tricks that were used, obstacles that were overcome, as well as share insights and lessons learned in the process of deploying puppet. Also covers our system for deploying templated jails around the world as part of our CDN and managing them with our Global Server Load Balancer (as discussed at EuroBSDCon 2012).

Highlights:

* What is puppet?
* Deploying puppetmaster for scale (using nginx, not passing large files through ruby)
* Managing config files
* Managing packages (with portupgrade)
* Advanced configuration files with templates
* Creating and using custom facts (freebsd specific facts)
* Deploying jails with puppet (with ezjail)
* Lessons Learned
    - Delivering large files requires some form of offloading
    - Templates are where the power is
    - Puppet is not like scripted deployment, manifests are different (and better)
* Where to go from here:
    - FreeBSD patches for facter, some of our custom facts should be standard
    - stored configs or puppetdb (needs porting), letting hosts know about each other
    - using puppet to automatically configure nagios